Modeling the Training Iteration Time for Heterogeneous Distributed Deep Learning Systems

Distributed deep learning systems effectively respond to the increasing demand for large-scale data processing in recent years. However, significant investment building distributed with powerful computing nodes places a huge financial burden on developers and researchers. It will be good predict precise benefit, i.e., how many times of speedup it can get compared training single machine (or few), before actually such big systems. To address this problem, paper presents novel performance model iteration time heterogeneous based characteristics parameter server (PS) system bulk synchronous parallel (BSP) synchronization style. The accuracy our is demonstrated by comparing real measurement results TensorFlow when different neural networks various kinds hardware testbeds: prediction higher than 90% most cases.